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SUMMARY 


Problem 


The  Naval  Submarine  Training  Cencer,  Pacific  DetachoenC  haa  expreaaed 
an  urgent  need  for  reliable,  objective  performance  meaaurement  of  aubmarlne 
fire  control  parties  while  they  are  operating  the  advanced  submarine  attack 
trainer,  21A40.  Existing  training  la  evaluated  primarily  by  experienced 
observers  who  judge  the  performance  of  the  approach  officer  (AO)  and  the 
fire  control  party  by  the  accuracy  of  their  target  motion  analysis  and  by 
weapon  hit  or  miss.  A method  is  needed  for  providing  objective  feedback 
%rlth  some  qualitative  performance  measure. 

Objective 


The  objectives  of  this  effort  were  to  design  a performance  evaluation 
device,  to  Install  It  within  the  advanced  submarine  attack  trainer,  and 
to  explore  Its  usefulness.  The  evaluation  device  was  to  use  real-time  in- 
formation and  to  operate  independently  of  the  main  tactical  computer.  The 
output  of  the  device  was  to  serve  as  an  objective  measure  of  training  effec- 
tiveness that  would  enhance  current  methods  of  post-performance  evaluation. 

Approach 

Real-time,  tactical  parameters  such  as  target  course  and  range  were 
chosen  for  use  In  the  performance  display  system  so  that  performance  evalua- 
tion could  be  provided  In  terms  already  used  by  the  fire  control  party.  In 
addition,  probability  of  counterdetection  [P(CD)]  was  Included  as  a measure 
that  is  dependent  upon  both  the  technology  of  the  target  and  the  approach 
taken  by  the  AO.  This  measure  Is  frequently  overlooked  by  tactical  trainers, 
which  concentrate  exclusively  on  skills  of  target  motion  analysis  and  weapon 
employment . 

A desktop  computer  with  a graphic  screen  was  chosen  as  the  display 
device  since  Its  stand-alone  capability  minimized  Interference  with  the 
main  computer's  functions  of  problem  generation.  For  realism,  the  desk- 
top coiaputer  sampled  geographical  and  sonar  Information  from  the  main  com- 
puter via  a binary  Interface  bus.  An  Integrated-circuit  board  was  built 
to  latch  and  hold  the  Information  as  well  as  to  respond  to  desktop  computer 
handshake  instructions  and  address.  The  desktop  sampled  values  at  fixed 
Intervals  that  were  set  by  the  experimenter. 

Actual  shlp-to-target  range,  fire  control  system  (calculated)  range, 
and  the  difference  between  actual  and  calculated  range  were  plotted  as  a 
function  of  time.  Similar  profiles  were  plotted  for  target  course  and 
for  PCCD).  Copies  of  these  profiles  were  given  to  the  trainees  for  use  In 
follow-up  critiques  and  to  trainer-operating  personnel  for  their  own  evalua- 
tion. 

Conclusions 

The  profiles  were  easy  to  read  and  gave  the  trainees  a unique  view  of 
progress  during  a simulated  submarine  approach,  and  attack.  An  extended 
period  of  use  by  submarine  crews  operating  the  trainer  will  be  necessary 
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INTRODUCTION 


Problem 


Submarine  tactical  team  training,  as  performed  in  shore-based  trainers, 
can  be  more  effective  if  it  Incorporates  objective  measures  of  system  per- 
formance in  addition  to  the  dichotomous  measures  (hit  or  miss)  currently 
being  used.  Trainers  capable  of  producing  realistic  sonar  activity  offer 
an  opportunity  to  Include  new  measures  based  on  self  noise,  target  noise, 
and  a variety  of  sound  transmission  conditions.  While  such  simulators 
can  generate  realistic  scenarios,  they  do  not  have  the  programming  flex- 
ibility needed  to  explore  a variety  of  information  displays  that  are 
candidates  for  performance  measurement.  Inexpensive  desktop  computers 
do  have  this  capability. 

Bac  kground 

Submarine  tactical  weapons  (usually  torpedos)  are  launched  following  an 
Intense  period  of  target  classification,  localization,  and  motion  analysis 
by  the  submarine  fire  control  team.  This  team,  under  the  direction  of  the 
approach  officer  (AO),  analyzes  sensor  information  and  maneuvers  the  ship 
until  fire  control  values  (i.e.,  target  course,  speed,  range,  and  depth) 
are  sufficiently  accurate  for  an  intercepting  weapon.  At  the  same  time, 
the  AO  must  avoid  counterdetection  and  possible  damage  to  his  own  ship. 

In  the  past,  the  submarine's  underwater  habitat  gave  it  an  invulner- 
ability that  made  counterdetection  unlikely  until  weapons  were  fired.  As 
long  as  certain  rules  were  observed,  such  as  avoiding  prolonged  periscope 
exposure  and  active  radar  or  sonar  transmission,  the  submarine  had  the 
tactical  advantage.. 

With  the  advent  of  nuclear  powered  submarines,  long-range  sonars,  and 
high-speed  sophisticated  weapons,  detection  and  classification  are  possible 
at  greater  distance  than  ever  before.  Consequently,  submarine-versus-sub- 
marine  warfare  demands  increased  emphasis  on  avoiding  counterdetection 
in  tactics  and  tactical  training.  The  new  weapons  also  Impose  probabilistic 
factors  on  the  tactical  decision  process  because  visual  confirmation  of 
target  classification  and  motion  is  unlikely.  New  tactics  are  being  de- 
veloped to  take  into  account  the  modern  sensors  and  weapons. 

Tactical  training  has  been  changing  in  step  with  tactical  development, 
but  problems  and  deficiencies  exist.  Sldorsky  and  Simoneau  (1970)  emphasized 
the  need  for  more  specific  feedback  in  performance  evaluation  than  the  gross 
hit-or-miss  measure  that  is  frequently  employed.  They  also  stressed  a need 
for  individual  decision-making  training  and  training  under  time  constraint. 
Pesch,  Hammell,  and  Ewalt  (1974)  reiterated  these  needs  and  recommended 
large-scale  decision-training  systems;  Hammell,  Gasteyer,  and  Pesch  (1973) 
recommended  modifications  to  existing  systems  for  Improved  performance 
evaluation  and  feedback.  Finally,  Hammell  et  al.  (1973)  and  Hammell, 

Sroka,  and  Allen  (1971)  stressed  providing  team  trainers  with  operational 
and  time  fidelity,  automatic  performance  measures,  and  interactive  opposi- 
tion. 


The  submarine  force  has  recently  installed  in  San  Uiego  a new  tactical 
team  trainer,  which  is  designed  to  provide  realistic,  computer-driven  tactical 
scenarios.  This  device,  the  21A40  Advanced  Submarine  Attack  Trainer,  gener- 
ates tactical  interactions  between  submarines  and  surface  craft  or  other 
submarines  via  a computer  that  incorporates  realistic  sound  transmission 
qualities.  It  is  part  of  a gradual  upgrading  of  submarine  tactical  train- 
ing equipment,  which  is  an  outgrowth  of  more  modern  ships  and  weapons. 

The  21A40  trainer  is  used  for  team  training  of  command  and  control, 
sonar,  and  fire  control  personnel  from  STURGEON  (SSN  637)  and  SCAMP  (SSN  388) 
class  submarines.  It  accurately  simulates  dynamic  ocean  environments  and 
acoustic  conditions,  own  ship  motion  control,  and  both  weapons  and  target 
characteristics. 

Targets,  weapons,  and  own  ship  can  be  assigned  to  independent  acoustic 
processing  channels,  which  attenuate  nolae  levels  as  a function  of  range 
and  bathythermal  conditions.  Thus,  the  physical  conditions  of  submarine 
sonar  detection  and  tracking  can  be  simulated  with  fidelity.  However,  there 
is  no  provision  for  displaying  Information  for  performance  evaluation  other 
than  by  replaying  a scenario,  or  for  providing  interactive  opposition  other 
than  through  computer-operator  intervention. 

Therefore,  the  Naval  Submarine  Training  Center,  Pacific  Detachment 
(NSTCPAC) , San  Diego  requested  that  the  Chief  of  Naval  Operations  (OP-29) 
consider  incorporating  a method  of  performance  evaluation,  tactical  decision- 
making training,  and  Interactive  opposition  (smart  target)  in  the  San  Diego 
21A40  trainer.*  The  Navy  Personnel  Research  and  Development  Center  made  a 
preliminary  response  to  this  request  and  took  steps  to  examine  the  feasibility 
of  providing  these  capabilities  by  a desktop  computer  that  would  use  infor- 
mation from  the  main  computer  without  changing  or  interfering  with  its  oper- 
ation and  configuration. 

Ob.lectlve 

The  developmental  work  reported  here  was  intended  to  meet  the  following 
goals: 

1.  To  determine  the  feasibility  of  sampling  real-time  information  from 
a tactical  team  trainer  for  storage  and  subsequent  feedback. 

2.  To  provide  a display  of  the  information  in  a form  useful  to  the 
trainer  operators  for  evaluating  training  sessions,  and  to  the  trainees 
for  evaluating  team  progress. 


^Naval  Submarine  Training  Center,  Pacific  Detachment,  San  Diego 
Itr  1500  Ser  127-75,  10  July  1975. 
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APPROACH 


The  proficiency  of  tactical  teams  historically  has  been  difficult  to 
assess  objectively.  One  reason  for  this  is  that  the  relationship  between 
the  outputs  of  the  overall  tactical  system  and  of  the  individual  system 
components  is  not  well  understood.  If  this  relationship  is  to  be  adequately 
specified,  the  behaviors  that  combine  to  determine  overall  system  response 
must  be  studied. 

It  is  generally  recogni^.ed  that  the  ultimate  index  of  tactical  effec- 
tiveness— weapon  hit  or  miss — is  not  sufficiently  diagnostic  to  provide 
data  useful  for  performance  evaluation.  Data  are  needed  that  are  more 
sensitive  to  variations  in  system  output.  Since  such  data  are  coincident 
with  the  activities  conducted  in  support  of  weapon  deployment,  they  are 
readlxy  available  during  training  exercises  involving  the  21AA0  simulator. 

In  the  present  study,  target  motion  parameters  calculated  by  the  fire 
control  team  throughout  the  duration  of  the  training  exercises  were  used 
as  data.  A preliminary  analysis  of  these  parameters  suggested  that  cal- 
culated target  range  and  course  were  particularly  useful  for  post  hoc 
evaluation  of  tactical  performance.  In  fact,  the  evaluation  method 
currently  used  in  training  relies  heavily  on  these  Indices  to  detect 
inappropriate  tendencies  in  a team's  analysis  of  target  motion. 

In  addition  to  quickly  and  accurately  determining  the  position  and 
motion  of  the  target,  the  approach  officer  (AO)  must  maintain  a posture 
that  minimizes  the  probability  of  his  ship  being  detected  by  the  target. 
Consequently,  the  probability  of  counterdetection  was  considered  along  with 
the  calculated  range  and  course  of  the  target  in  evaluating  tactical  team 
performance.  In  this  way,  it  was  believed  that  a more  complete  represen- 
tation of  the  actual  target  engagement  situation  could  be  expressed  and 
chat  an  important  tactical  factor  could  be  added  to  the  evaluation  process. 

It  was  highly  desirable  that  Che  data  collection  to  support  this  evalua- 
tion process  not  interfere  with  either  the  21A40  trainer  or  the  training 
procedure  itself.  As  a result,  data  collection  and  summarization  procedures 
were  designed  to  function  within  a flexible  module  independently  of  the 
operation  of  both  Che  main  21AA0  computer  and  typical  training  activities. 
Thus,  the  entire  process  supporting  AO  performance  evaluation  was  intended 
to  be  as  unobtrusive  as  possible. 

Scenarios 

The  21A40  trainer  has  been  primarily  employed  in  developing  and  in 
maintaining  advanced  tactical  skills,  although  it  has  also  been  exploited 
in  training  related  skills,  such  as  navigation.  Numerous  scenarios  have 
been  developed  for  use  with  the  21A40  trainer  in  order  to  exercise  tactical 
circumstances  likely  to  be  encountered  by  an  AO.  These  scenarios  Involve 
various  combinations  of  enemy  surface  and  subsurface  vessels  opposing  own 
ship  (l.e.,  an  SSN  588  or  SSN  637  class  submarine).  The  behavior  of  the 
enemy  ships  is  prespecified  and  is  dependent  on  the  elapsed  time  of  the 
exercise.  Departures  from  the  scenario  script  are  made  at  the  discretion 
of  the  training  officer  on  the  basis  of  own  ship's  activity  and  of  particular 
Instructional  needs.  In  the  present  study,  no  attempt  was  made  to  alter 
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the  normal  course  of  the  exercise.  Data  were  gathered  during  scenarios 
varying  considerably  In  the  composition  of  the  enemy  force.  Surface  con- 
voys, Individual  submarines,  and  surface  convoys  supported  by  submarines 
were  Included. 

Participants 

During  the  period  from  July  to  October  1977,  data  from  three  actual 
submarine  tactical  teams  using  the  21A40  simulator  were  collected.  In 
addition,  performance  was  measured  during  exercises  constructed  for 
training  prospective  submarine  commanding  officers. 

Equipment  Description 

The  Tektronix  4051  graphics  terminal  was  selected  to  support  the  tac- 
tical performance  evaluation  process  developed  here  on  the  basis  of  Its 
extreme  versatility,  compact  size,  and  widespread  availability  throughout 
the  submarine  force.  The  Tektronix  4051  Is  a desktop  computer  with  ex- 
tensive graphics  capabilities.  It  is  fully  programmable  in  a modified 
version  of  BASIC,  which  includes  extensions  to  support  the  graphics  func- 
tions. Programs  and  data  may  be  stored  on  and  retrieved  from  an  associated 
magnetic  tape  cartridge.  In  addition,  the  4051  may  be  readily  interfaced 
with  other  computers  via  the  data  communication  interface  or  with  miscel- 
laneous peripheral  devices  via  the  general  purpose  interface  bus  (GPIB) . 

A critical  issue  in  instrumenting  an  AO  performance  evaluation  pro- 
cedure concerns  the  availability  of  relevant  data.  Target  motion  analysis 
and  sonar  data  are  resident  in  the  21A40  trainer,  which  simulates  the 
ocean  environment  and  which  is  controlled  by  the  UYK-7  computer.  Since 
the  4051  was  intended  as  the  vehicle  for  the  performance  evaluation,  an 
effective  interface  between  these  computers  was  necessary.  The  GPIB  was 
chosen  as  the  means  by  which  to  accomplish  this  interface  with  minimal 
modifications  to  the  21A40  system. 

Interface  Design 

To  maximize  the  ease  of  use  and  the  flexibility  of  the  interface  be- 
tween the  4051  GPIB  and  the  21A40,  a design  was  developed  to  conform  to 
IEEE  Standard  488-1975.  A schematic  diagram  of  an  interface  designed  for 
a similar  application  was  acquired  from  the  Scripps  Institute,  San  Diego 
and  was  examined  within  the  21A40  system.  Significant  revision  of  this 
initial  design  was  required  to  permit  multiple  8-bit  bytes  to  be  concurrently 
sampled  from  the  21A40  and  to  eliminate  a considerable  amount  of  extraneous 
circuitry.  A schematic  diagram  of  the  resulting  circuitry  is  provided  in 
Appendix  A.  Although  this  circuit  is  potentially  capable  of  sampling  numerous 
8-bit  data  bytes,  it  is  presently  designed  to  accommodate  only  the  five  bytes 
representing  the  data  used  in  this  study. 

Some  modification  to  both  the  hardware  and  the  software  of  the  21A40 
trainer  was  required  to  support  the  interface  with  the  4051  GPIB.  These 
modifications  are  documented  in  Appendices  B and  C,  respectively.  Basically, 
these  alterations  involve  directing  the  specified  data  to  a circulating 
storage  card  that  resides  in  the  21A40.  From  this  card,  the  data  are 
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serially  loaded  into  particular  locations  of  two  discrete  storage  cards 
that  are  wired  to  the  GPIB  interface  circuitry.  The  contents  of  these 
discrete  storage  locations  are  updated  each  second  and  are  transferred  to 
the  GPIB  only  when  addressed  by  the  4051.  Consequently,  the  4051  may  access 
valid  data  at  any  desired  sampling  rate. 

Data  Collection  Software 

A program  was  developed  to  run  on  the  4051  that  would  automatically 
collect  and  store  data  relevant  to  the  AO's  tactical  performance.  The 
data  used  in  this  study  were  (1)  the  true  range  and  course  of  the  target, 

(2)  the  range  and  course  of  the  target  as  calculated  by  the  fire  control 
party,  and  (3)  the  signal/noise  ratio  for  ship  as  measured  at  the  target. 
These  data  were  provided  to  the  4051  via  the  GPIB  interface,  which  is  dis- 
cussed above  and  is  addressed  as  peripheral  device  40.  A complete  listing 
of  this  program  is  provided  in  Appendix  D. 

Although  this  data  collection  program  functions  automatically,  some 
operator  Intervention  is  required,  particularly  during  initialization.  A 
set  of  instructions  that  describes  how  to  use  this  tactical  data  collection 
program  has  been  prepared  and  appears  in  Appendix  E.  After  the  program  is 
loaded  into  the  4051,  an  operator  is  required  to  specify  several  parameters 
and  to  select  a data  sampling  rate  from  a menu  that  appears  on  the  screen. 

In  addition,  he  must  provide  the  program  with  an  appropriate  indication 
of  the  target  ship's  passive  sonar  sensitivity  by  entering  estimated  values 
for  the  mean  and  standard  deviation  of  the  target  ship's  recognition  dif- 
ferential. Although  these  parameters  could  have  been  automatically  selected 
on  the  basis  of  target  ship  classification,  sonar  sensitivity  information 
was  not  stored  in  order  to  keep  the  program  Itself  unclassified. 

Once  these  parameters  have  been  entered  and  the  scenario  has  been 
initiated,  the  program  may  be  left  unattended  until  the  termination  of 
the  training  exercise.  At  this  point,  the  operator  should  press  a special 
function  key  to  signal  the  end  of  the  exercise.  Another  function  key, 
which  has  been  provided  to  act  as  an  event  marker,  may  be  used  to  mark 
the  time  of  occurrence  of  any  significant  event,  such  as  weapon  deploy- 
ment. 

Throughout  the  training  exercise,  a status  display  of  the  AO's  tactical 
performance  data  is  provided  on  the  4051 's  screen.  This  feature  enables 
transient  observers  to  review  momentary  fluctuations  in  the  tactical  team's 
performance  while  the  scenario  is  still  in  progress.  Figure  1 provides 
an  Illustration  of  this  status  display. 
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Figure  1.  Status  display  of  tactical  performance  data. 


Data  Summarization  Software 


Following  the  conclusion  of  a training  exercise,  the  operator  Is  given 
the  option  to  Immediately  rerun  the  data  collection  sequence,  to  examine 
the  data  just  gathered,  or  to  terminate  all  subsequent  activity.  Under 
most  circumstances,  the  operator  will  elect  to  examine  the  data.  In  this 
case,  the  4051  Is  automatically  routed  to  another  program  that  retrieves 
the  performance  data  and  plots  profiles  of  these  data  across  the  period 
of  the  exercise.  A listing  of  this  program,  which  may  be  accessed  Inde- 
pendently of  the  data  collection  program.  Is  provided  In  Appendix  F. 

At  the  request  of  the  operator,  this  program  will  retrieve  and  display 
data  about  target  course,  target  ranpe,  or  probability  of  counterdetection 
[P(CD)].  For  target  course  and  range,  both  absolute  and  difference  plots 
of  the  actual  and  calculated  values  are  presented.  As  seen  In  Figures  2 
and  3,  which  Illustrate  typical  performance  profiles  for  target  course  and 
range  respectively,  both  of  these  profiles  are  plotted  as  a function  of 
time.  The  scale  of  these  plots  Is  varied  automatically,  depending  upon 
the  numbers  of  samples  taken  and  the  data  values  observed.  In  the  upper 
plots,  the  solid  line  shows  the  true  behavior  of  the  target,  and  the  dotted 
line  reflects  the  calculations  by  the  fire  control  team.  The  lower  plots 
are  simply  the  differences  between  these  data;  that  is,  the  target  motion 
analysis  error.  The  upwards  pointing  arrow  is  an  event  marker,  which,  in 
this  case,  reflects  the  firing  of  a torpedo  salvo. 

In  addition  to  the  target  course  and  range  profiles,  fluctuations 
In  P(CD)  throughout  the  training  exercise  may  be  displayed.  As  shown  In 
Figure  4,  which  presents  a typical  P(CD)  profile,  the  basic  structure  of 
these  plots  Is  similar  to  those  for  course  and  range.  However,  the  upper 
plot  In  Figure  4 depicts  the  slgnal/nolse  (actually  signal  excess)  condi- 
tions as  measured  at  the  target  ship  throughout  the  scenario.  The  lower 
plot  Is  simply  a Gaussian  transformation  of  the  slgnal/nolse  profile  that 
permits  P(CD)  to  be  more  readily  observed: 

2 

P(CD)  - .5  + .5  (1  - e~^^ 
where  x ■ signal  excess 

One  of  the  anticipated  benefits  of  these  performance  profiles  Involves 
the  fact  that  they  are  almost  immediately  available  following  the  completion 
of  a training  exercise.  Consequently,  they  may  be  used  in  the  critique 
that  Is  typically  held  after  the  close  of  each  scenario.  To  promote  their 
use  by  all  participants  In  these  critiques,  hard  copies  of  each  performance 
profile  for  the  preceedlng  training  session  were  provided  to  the  trainees 
as  well  as  to  the  training  personnel.  These  copies  were  made  using  a hard 
copy  unit  that  Is  compatible  with  the  4051,  such  as  the  Tektronix  4631. 
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Figure  2.  Target  course  performance  profile. 
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Figure  3.  Target  range  performance  profile 
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Figure  4.  Probability  of  counterdetectlon  profile. 


findings 


It  should  be  emphasized  that  the  present  examination  of  procedures 
for  measuring  the  proficiency  of  the  approach  officer  (AO)  is  preliminary. 
The  performance  profiles  used  here  are  Intended  as  a first  step  towards  the 
development  of  a meaningful  scheme  for  performance  evaluation.  Ultimately, 
advances  In  evaluating  submarine  AO  performance  are  expected  also  to  provide 
Information  about  ways  to  adaptively  augment  training  by  the  use  of  smart 
(interactive)  targets  and  to  aid  the  AO  In  making  tactical  decisions. 

Instrumentation 


A major  goal  of  the  present  effort  was  to  Implement  an  accurate  and 
reliable  means  for  gathering  data  relevant  to  the  AO's  tactical  performance. 
The  Interface  that  was  developed  to  link  the  4051  and  the  21A40  has  provided 
this  capability.  Since  Its  implementation,  no  Incorrect  data  have  been 
transferred  to  the  4051.  However,  since  the  data  bytes  are  expressed  In 
only  8-blts,  some  Imprecision  Is  Introduced.  Target  range  and  course  data 
are  accurate  within  125  yards  and  1.4°,  respectively.  Exact  slgnal/noise 
data  are  transferred.  The  data  collection  Interface  required  minimal 
modification  to  the  existing  21A40  system  and  operates  almost  entirely 
unobtrusively. 

The  use  of  the  4051  as  an  Intelligent,  external  module  to  the  21A40 
trainer  enables  the  system  to  be  extremely  flexible.  Modifications  to 
the  existing  performance  measurement  procedures  may  be  made  easily,  and 
new  features  may  be  readily  adopted  and  tested. 

One  limitation  concerning  the  operation  of  this  data  collection  inter- 
face has  been  encountered.  Due  to  restrictions  within  the  21A40  system, 
the  data  collection  Interface  may  only  be  activated  when  the  trainer  is 
in  combined  mode.  That  is,  to  get  meaningful  probability  of  counterdetec- 
tion (P(CD))  data,  the  two  attack  centers  within  the  trainer  must  be  linked 
to  the  same  scenario.  Consequently,  P(CD)  data  cannot  presently  be  col- 
lected while  both  attack  centers  in  the  21A40  trainer  are  operating  inde- 
pendently. 

Usefulness  In  the  Advanced  Submarine  Attack  Training  Environment 

The  Informal  reactions  from  the  trainers  and  trainees  who  have  used 
the  tactical  performance  profiles  developed  here  have  been  favorable. 

Numerous  tendencies  that  characterize  a tactical  team's  performance  may 
be  detected  easily  and  examined  quantitatively.  It  is  clear,  however,  that 
the  value  of  this  method  of  performance  evaluation  must  be  determined  as  a 
result  of  an  extended  period  of  use  by  submarine  crews  operating  the  trainer. 

An  Illustration  of  the  types  of  Information  that  may  be  accumulated 
via  this  procedure  may  be  seen  In  Figures  2,  3,  and  4.  The  scenario  rep- 
resented by  these  figures  Involved  an  SSN  637  as  own  ship  versus  a snorkel- 
Ing  FOXTROT-class  Soviet  submarine.  The  peak  In  the  true  course  shown 
in  the  upper  plot  of  Figure  2 reflects  a circular  movement  by  the  target. 
Se/eral  typical  performance  characteristics  may  be  observed  In  Figures  2 
and  3.  A rather  gratifying  tendency  towards  a small  error  In  the  calculated 
target  parameters  may  be  seen  to  develop  as  the  exercise  proceeds.  At  the 
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point  of  weapon  deployment,  signified  by  the  arrow,  the  fire  control  team's 
range  and  course  calculations  are  quite  accurate.  Another  general  tendency 
that  may  be  seen  in  these  figures  Involves  the  discrete,  oscillatory  nature 
of  the  adjustments  that  the  fire  control  team  make  in  their  target  motion 
analysis.  This  behavior  appears  to  be  a result  of  standard  procedures  and 
of  the  time  delays  Inherent  in  the  analysis  of  target  motion. 

The  signal/noise  profile  in  Figure  A reflects  the  sound  transmission 
from  own  ship  to  the  target.  Throughout  most  of  the  exercise,  the  sound 
conditions  were  relatively  stable,  resulting  in  a very  small  P(CD).  How- 
ever, about  two-thirds  of  the  way  through  the  scenario,  a relatively  small 
Increase  in  the  slgnal/nolse  may  be  seen  to  result  in  a dramatic  Increase 
in  P(CD).  At  this  point,  the  AO  was  unaware  that  his  posture  relative  to 
that  of  the  target  could  have  resulted  in  a counterdetection.  Such  infor- 
mation has  certainly  been  of  assistance  in  advising  AOs  of  the  consequences 
of  their  own  maneuvers.  The  precipitous  drop  in  signal/noise  that  may  be 
noted  in  Figure  A is  a result  of  own  ship  moving  into  the  baffles  of  the 
target.  These  marked  discontinuities  are  clearly  apparent  in  the  data  and 
are  indicative  of  the  use  of  advantageous  tactics. 

These  observations  are  suggestive  of  several  approaches  that  might  be 
taken  to  develop  more  unitary  and  diagnostic  indices  of  tactical  proficiency. 
In  particular,  useful  composite  performance  measures  might  be  devised  based 
on  such  factors  as  (1)  the  time  for  the  fire  control  team  to  recover  from 
a target's  change  in  course,  (2)  the  time  to  achieve  an  overall  target 
motion  solution  that  lies  within  the  corrective  tolerance  of  the  weapon, 

(3)  the  oscillation  magnitude  of  subsequent  target  motion  solutions,  and 
(A)  the  level  of  P(CD).  The  most  likely  possibility  for  the  development 
of  such  a composite  measure  of  tactical  team  performance  involves  the 
development  of  a multiple  regression  model.  In  any  case,  a considerable 
amount  of  data  from  a wide  variety  of  scenarios  must  be  gathered  to  support 
the  construction  of  such  a performance  evaluation  index.  The  general  pro- 
cedure that  has  been  described  in  this  report  enables  such  a data  base  to 
be  easily  accumulated  and  Interrogated. 

In  exercising  the  present  AO  performance  evaluation  process,  several 
limitations  have  become  apparent.  In  particular,  the  current  procedure 
is  capable  of  dealing  with  the  data  from  only  one  target.  In  many  scenarios, 
it  is  entirely  reasonable  to  require  data  from  several  targets  to  be  gathered 
simultaneously.  For  instance,  if  own  ship  encountered  an  enemy  convoy,  the 
P(CD}  from  several  escorts  should  be  monitored  as  the  AO  maneuvers  his 
submarine.  As  subsequent,  more  sophisticated  performance  evaluation  pro- 
cedures are  developed,  the  ability  to  handle  multiple  targets  will  certainly 
be  addressed. 

Another  limitation  which  has  been  observed  concerns  the  usefulness  of 
P(CD)  as  a performance  measure.  Due  to  the  Inherent  sonar  sensitivities 
and  noise  emission  patterns  of  surface  ships,  U.  S.  Navy  submarines  are 
normally  quite  difficult  to  detect.  Consequently,  the  dlagnostlclty  of 
P(CD)  is  largely  constrained  to  passive  sonar  submarine-submarine  encounters. 
Although  this  is  in  Itself  a valuable  contribution,  another  means  should  be 
developed  for  evaluating  own  ship  posture  during  encounters  with  surface 
ships  with  active  sonars. 
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CONCLUSIONS 

The  procedure  and  supporting  instrumentation  reported  here  has  been 
demonstrated  to  provide  an  effective  means  for  tactical  performance  measure- 
ment of  the  approach  officer  (AO)  and  the  fire  control  team.  However,  the 
extent  to  which  the  performance  evaluation  system  influences  tactical  train- 
ing remains  to  be  seen.  Regular  use  under  a variety  of  scenarios  will  pro- 
vide a basis  for  revisions  to  and  subsequent  extensions  of  the  present  pro- 
cedure. As  such  changes  are  indicated,  they  may  be  readily  and  inexpensively 
incorporated. 

Once  a particular  data  set  and  display  format  are  adopted,  they  may 
be  examined  for  dlagnostlclty  under  a wide  variety  of  tactical  scenarios. 
Examples  of  data  with  potential  diagnostic  value  include:  (1)  time  to 
arrive  at  a weapon-compatible  solution  without  exceeding  a prespecified 
counterdetection  probability,  (2)  length  of  time  an  AO  can  maintain  a low 
counterdetection  profile  while  remaining  within  weapon  range  and  maintaining 
satisfactory  target  motion  analysis,  and  (3)  time  interval  between  target 
course  or  speed  change  and  satisfactory  motion  analysis  (i.e.,  zig  recovery 
time).  These  data  may  be  combined  into  a polynomial  expression  to  provide 
a composite  indication  of  tactical  proficiency. 

These  same  data  also  may  offer  a means  of  providing  more  interactive 
opposition  during  advanced  tactical  training.  Particularly  through  the 
probability  of  counterdetection  (P(CD))  measure,  it  is  possible  to  trigger 
changes  in  target  maneuvers  and  intentions.  Given  an  adequate  P(CD),  a 
smart  target  could  be  programmed  to  evade  or  counterattack.  Changes  in 
target  actions  thus  could  be  substantiated  by  the  posture  adopted  by  the 
AO,  his  tactical  skills,  and  the  proficiency  of  the  fire  control  team. 

The  modular  structure  of  the  present  procedure  has  several  advantages 
in  system  design  and  flexibility.  It  may  be  readily  expanded  to  Include 
new  developments  in  training  evaluation,  such  as  those  described  above, 
or  to  support  additional  equipment  without  disturbing  other  segments  of 
the  system.  It  also  might  be  Incorporated  into  submarine  attack  trainers 
other  than  the  21A40.  Conversely,  it  could  also  be  disconnected  from  the 
system  with  minimal  expense  and  time.  The  portability  of  this  performance 
measurement  module  also  permits  Its  use  at  remote  sites.  For  Instance, 
it  might  be  used  on  board  submarines  with  digital  computer  fire  control 
systems,  such  as  the  MK  117. 
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RECOMMENDATIONS 

Based  on  the  results  of  this  study,  the  following  recommendations  are 
made: 

1.  Development  of  this  evaluation  system  should  be  continued,  directing 
future  efforts  toward  examining  data  for  dlagnosticlty  under  a variety  of 
tactical  scenarios.  From  these  data,  a composite  Indicator  of  tactical  pro- 
ficiency may  emerge. 

2.  The  use  of  probability  of  counterdetection,  in  conjunction  with  the 
target  motion  analysis  parameters,  should  be  examined  as  a basis  for  pro- 
gramming adaptive  targets. 
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APPENDIX  A 

INTERFACE  SCHEMATIC  DIAGRAM 


Figure  A-1.  Schematic  diagram  of  GPIB  interface  card. 
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APPENDIX  B 

DESCRIPTION  OF  HARDWARE  MODIFICATIONS 
TO  THE  21A40  TRAINER 
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DESCRIPTION  OF  HARDWARE  MODIFICATIONS 
TO  THE  21A40  TRAINER 


The  Tektronix  4051  samples  21A40's  real-time  program  by  fetching  fire 
control  range  and  course,  true  range  and  course,  and  signal-to-noise  ratio 
of  any  target  specified  with  the  21A40  operator  command  PDl,  PD2,  PD3,  etc. 
up  to  PD8.  These  data  are  first  stored  in  the  acoustic  data  processor's 
(ADP)  self-band  amplitude  (SBA)  output  block.  The  five  8-bit  words  are 
then  tapped  off  of  ADP  data  fan-out  at  the  input  of  card  9A1A2A117  (Figure 
B-1),  sent  to  an  existing  circulating  storage  card  9A1A2A32S  (Figure  B-2), 
and  latched  up  with  load  pulse  115,  which  loads  64  words  of  SBA  data. 
(PROBDET  data  is  loaded  in  byte  3 of  SBA,  which  is  unused.)  The  data  is 
then  stored  in  two  new  discrete  storage  cards,  9A1A2A101  and  9A1A2A116, 
each  of  which  is  capable  of  storing  four  8-blt  words.  From  timing  distri- 
bution 9A1A2A417  (Figure  B-3),  word  times  are  routed  to  the  two  new  discrete 
storage  cards  (Figures  B-4  and  B-5). 

Word  time  55  in  the  hardware  corresponds  to  the  first  word  stored 
in  ADP's  SBA  output  block;  word  time  56,  to  the  second  word;  word  time 
57,  to  the  third  word,  etc.  up  to  word  time  59,  which  corresponds  to  the 
last  word  in  the  output  block.  The  first  four  words  go  to  discrete  storage 
9A1A2A101,  and  the  last  word  time  goes  to  discrete  storage  9A1A2A116. 

fThe  PROBDET  data  is  then  left  hanging  on  the  outputs  of  the  discrete  storage 

cards  (four  8-bit  bytes  on  one  card,  and  one  8-bit  byte  on  the  other).  Ad- 

! dltional  details  on  Figures  B-1  through  B-5  may  be  found  in  NAVTRADEV  Publi- 

t ■ cation  No.  4014,  1975. 

( 

The  output  of  the  discrete  storage  cards  (In  bytes)  are  then  connected 
to  an  Interface  card  (9A1A2A412  in  Appendix  A),  which  receives  up  to  five 
bytes  of  data,  and  serially  transfers  the  data  to  a Tektronix  4051  desk- 
top computer  for  analysis  during  the  running  of  a problem.  At  the  end  of 
a problem,  profiles  of  the  differences  between  the  true  and  the  fire  con- 
trol solution  for  range  and  for  course  are  plotted  for  performance  during 
that  exercise.  In  addition,  similar  profiles  of  slgnal-to-nolse  ratio  and 
probability  of  counterdetection  were  displayed  for  use  in  the  performance 
critique. 
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Figure  B-1.  Block  diagram  of  21A40  connections  fcr  Prob.  Det.  word  transfer 


Figure  b-2 . Jetail  of  connections  between  sonar  echo  generator  and  circulating  storage 
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APPENDIX  C 

SOFTWARE  MODIFICATIONS  TO  THE  21A40  TRAINER 
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Figure  C-2.  Prob.  det.  transvector  (TV)  routine. 
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Figure  C-3.  Prob.  det.  data  acquisition  routine. 
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APPENDIX  D 

4051  BASIC  PROGRAM  FOR  AUTOMATIC  DATA  COLLECTION  AND  STORAGE 
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730  PRINT  "S/N  » "iNlf-P(CD)  « •' | PI , "SAMPLE  # * "IX 

740  PRI  "saascssassaxaasssazssasasxsaxssssassassassa; 

750  PRINT  033:R1|R2|D1}D2|N1|P1 
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The  Tektronix  4051  is  being  used  as  part  of  a system  to  help  evaluate 
the  tactical  performance  of  the  approach  officer  (AO)  and  the  fire  con- 
trol team  during  training  exercises  in  the  21A40.  To  support  this  evalua- 
tion process,  the  4051  automatically  samples  range,  course,  and  signal/ 
noise  data  from  the  21A40.  However,  before  this  process  may  begin,  both 
the  21A40  and  the  4051  must  be  initialized.  A few  preparatory  activities 
are  required  to  complete  this  Initialization. 

Selecting  the  Training  Problem 

During  typical  usage  of  this  system,  the  training  officer  selects  the 
problem  to  be  used.  The  parameters  of  the  training  problem,  which  specify 
the  own  ship,  target,  and  ocean  conditions,  are  documented  and  are  entered 
into  the  21A40  via  standard  procedures  using  one  of  the  associated  terminals. 
These  procedures,  along  with  representative  values  of  the  scenario  para- 
meters, are  outlined  below. 

1.  Specify  the  type  of  own  ship  to  be  used  as  well  as  its  speed, 
course,  and  depth.  By  entering  each  of  the  following  commands  sequentially, 
own  ship  will  be  identified  as  an  SSN  637  moving  at  a velocity  of  10  knots, 
on  a course  of  090®,  and  at  a depth  of  190  feet. 


a. 

OSl, 

TYP,  SSN 

b. 

OSl, 

VEL,  10 

c. 

OSl, 

CS,  090 

d. 

OSl, 

D/A,  190 

e. 

OSl, 

A/D 

2.  Specify  the  type  of  target  to  be  used  as  well  as  its 
own  ship,  its  course,  and  its  speed.  The  following  commands 
a DD  1-class  target  at  an  initial  range  and  bearing  from  own 
yards  and  030®,  respectively.  This  target  will  be  traveling 
of  120®  at  a speed  of  10  knots. 


a. 

TGI, 

TYP,  DDl 

b. 

TGI, 

POO,  15000, 

c. 

TGI, 

CS,  120 

d. 

TGI, 

VEL,  10 

e. 

TGI, 

A/D 

position  from 
will  generate 
ship  of  15,000 
on  a course 


3.  Specify  the  type  of  ocean  to  be  used.  Ocean  conditions  are  deter- 
mined by  the  depth  and  type  of  bottom,  the  sound  velocity  profile  (which 
reflects  water  conditions  affecting  sound  transmission,  such  as  salinity, 
temperature,  etc.),  and  the  sea  state.  For  example,  a rock  bottom  at 
12,500  feet  with  a sound  velocity  profile  of  6 and  a sea  state  of  2 would 
be  activated  by  entering  the  following  commands. 


I 


E-l 
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a.  BMR,  12500 

b.  SVP,  6 

c.  SS,  2 

d.  GO 

Default  ocean  parameters  may  be  activated  by  simply  entering  the  command 
GO.  This  will  automatically  define  an  ocean  with  a rock  bottom  at  a depth 
of  25,000  feet  with  sound  velocity  profile  of  0 and  a sea  state  of  0. 

4.  Determine  which  target  will  be  analyzed  by  attack  center  1.  Enter 
PD  along  with  the  appropriate  target  number.  For  instance,  if  attack 
director  1 will  be  analyzing  target  1,  then  enter  PDl. 

Activating  the  4051  Data  Collection  Program 

1.  Turn  on  the  4051  by  pressing  the  switch  under  the  right  side  of 
the  keyboard. 

2.  Press  the  HOME/PAGE  key  In  the  upper  left  of  the  keyboard. 

3.  Firmly  press  the  tape  cartridge  Into  solt.  The  metal  side  should 
be  on  the  right  side. 

4.  Consult  the  Session  Log  located  near  the  4051  and  fill  in  the 
necessary  information  (date,  problem  number,  etc.). 

5.  Enter  the  number  of  the  upcoming  session  (from  the  Session  Log) 
into  the  4051  and  then  press  the  RETURN  key. 

6.  Select  a sampling  rate  from  those  shown  on  the  4051 's  screen  that 
seems  appropriate  to  the  upcoming  problem  (If  you  are  not  sure,  select 

30  seconds).  Enter  the  number  corresponding  to  this  sampling  rate  and 
press  RETURN. 

7.  In  response  to  the  question  "Has  this  file  been  marked?",  which 
will  appear  on  the  4051  screen,  type  NO  and  press  RETURN. 

8.  Enter  a recognition  differential  and  a standard  deviation  that 

are  appropriate  for  the  target’s  sonar  sensitivity  In  the  upcoming  problem. 
If  you  are  not  sure  of  appropriate  values,  consult  the  training  officer. 
Press  RETURN  after  entering  each  of  these  values.  For  example,  you  might 
enter  -10  and  then  press  RETURN,  followed  by  9 and  then  RETURN. 

Initiating  the  Problem 

1.  At  the  direction  of  the  training  officer.  Initiate  the  problem  by 
entering  RUN  on  a 21A40  terminal. 

2.  Make  sure  that  one  of  the  21A40  alphanumeric  terminals  Is  displaying 
PAGE  5. 


3.  When  the  problem  Is  underway  and  the  training  team  Is  active, 
press  REfURN.  This  will  start  the  data  collection  process.  Note  the 
problem  time  at  which  you  started  the  4051  and  record  It  on  the  Session 
Log. 
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Operating  the  Problem 


1.  During  the  problem,  target  motion  variations  may  be  made  via  the 
21A40  terminals  under  the  direction  of  the  scenario  plan  or  of  the  train- 
ing officer. 

2.  The  4051  data  collection  process  will  now  operate  on  its  own 
throughout  the  problem.  If,  at  any  point  during  the  problem,  you  would 
like  to  make  note  of  an  event  of  Interest,  press  button  //I,  labeled  "Event 
Mark,"  which  is  located  in  the  upper  left  of  the  keyboard.  It  is  highly 
recommended  that  you  do  this  when  the  training  team  fires  a torpedo,  but 
other  events  of  Importance  may  also  be  marked. 

3.  At  the  conclusion  of  the  problem,  press  button  //lO,  labeled  "End 

of  Data,"  which  is  located  in  the  upper  left  of  the  keyboard.  This  signals 
the  4051  that  the  problem  is  over.  You  will  now  be  asked  what  you  want 
to  do  at  this  point.  You  can  either  just  stop  altogether  (quit),  im- 
mediately collect  more  data  (rerun),  or  look  at  the  data.  Under  most 
circumstances,  you  should  look  at  the  data.  Enter  the  appropriate  letter 
for  one  of  these  actions  and  then  press  RETURN. 

Examining  the  Data 

1.  Turn  on  the  hard  copy  unit  that  is  next  to  the  4051. 

2.  Enter  the  number  of  the  session  that  you  want  to  examine,  and  then 
press  RETURN. 

3.  Enter  the  sampling  rate  used  in  the  session,  and  then  press  RETURN. 
For  example,  you  might  type  30  SEC.  This  will  be  used  as  a label  in  the 
graphs  that  will  be  plotted. 

4.  Select  the  type  of  data  that  you  want  to  see  displayed  (range, 
course,  or  detectability).  Enter  the  appropriate  letter  and  press  RETURN. 

5.  Make  a copy  of  the  graphs  shown  on  the  4051  screen  by  pressing  the 
button  labeled  MAKE  COPY,  located  in  the  upper  right  area  of  the  keyboard. 

6.  Press  RETURN  and  select  the  next  type  of  data  to  be  displayed 
(see  step  //4  above) . 

7.  Give  the  paper  copies  of  the  graphs  to  the  training  officer  for 
use  in  the  critique. 

8.  When  finished,  press  F followed  by  RETURN,  and  then  turn  off  both 
the  hard  copy  unit  and  the  4051.  If  you  want  to  collect  data  from  another 
probleu,  press  the  AUTO  LOAD  button  and  continue  as  before. 
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